What is the real-world question? This could be hypothesis-driven or discovery-based.
Why is this question interesting and important? What are the implications of better understanding this data?
Briefly describe any background information necessary to understand this problem.
Briefly describe how this question can be answered in the context of a model or analysis.
Outline the rest of the report/analysis.
2 Data
What is the data under investigation? Provide a brief overview/description of the data.
Describe how your data connects to the domain problem.
2.1 Data Collection
How was the data collected or generated (including details on the experimental design)? Be as transparent as possible so that conclusions made from this data are not misinterpreted down the road.
Describe any limitations when using the data to answer the domain problem of interest.
Where is the data stored, and how can it be accessed by others (if applicable)?
2.2 Data Splitting
TODO: add advice for possible data splits, AK getting nice figure together
Decide on the proportion of data in each split.
Decide on the “how” to split the data (e.g., random sampling, stratified sampling, etc.), and explain why this is a reasonable way to split the data.
Split the data into a training, validation, and test set.
Provide summary statistics and/or figures of the three data sets to illustrate how similar (or different) they are.
2.2.1 Data Splitting Overview
X Data Split
plot_X_datasplit(Xtrain, Xvalid, Xtest)
Y Data Split
plot_y_datasplit(ytrain, yvalid, ytest)
2.3 Data Cleaning and Preprocessing
What steps were taken to clean the data? More importantly, why was the data cleaned in this way?
Discuss all inconsistencies, problems, oddities in the data (e.g., missing data, errors in data, outliers, etc.).
Record your preprocessing steps in a way such that if someone else were to reproduce your analysis, they could easily replicate and understand your steps.
It can be helpful to include relevant plots that explain/justify the choices that were made when cleaning the data.
If more than one preprocessing pipeline is reasonable, examine the impacts of these alternative preprocessing pipelines on the final data results.
Again, be as transparent as possible. This allows others to make their own educated decisions on how best to preprocess the data.
2.4 Data Exploration
TODO: Add drag and drop feature in shiny version for other images
The main goal of this section is to give the reader a feel for what the data “looks like” at a basic level.
Provide plots that summarize the data and perhaps even plots that convey some smaller findings which ultimately motivate the main findings.
Provide additional plots representing remaining oddities after pre-processing if applicable.
Add summary statistics in accompanying tables (or in figures) for quick comparisons.
2.4.1 Training Data Overview
#> Number of features: 4
#> Number of training samples: 90
#> Number of NAs in training y: 0
#> Number of NAs in training X: 0
#> Number of columns in training X with NAs: 0
#> Number of constant columns in training X: 0
Summary Tables
# summary of types of features in training Xdata_types(Xtrain = Xtrain, ytrain = ytrain)
# summary of features in training X, sorted by typedt_ls <-data_summary(Xtrain = Xtrain, ytrain = ytrain, digits =2, sigfig = F)for (dt_name innames(dt_ls)) { simChef:::subchunkify(dt_ls[[dt_name]], i = chunk_idx, other_args ="results='asis'") chunk_idx <- chunk_idx +1}
X distribution
# plot X distributionplot_X_distribution(Xtrain, "density")
Y distribution
# plot y distributionplot_y_distribution(ytrain, "bar")
TODO: add advice on which models to select and why
Discuss the prediction methods under consideration, and explain why these methods were chosen.
Discuss the accuracy metrics under consideration, and explain why these metrics were chosen.
Note: there should be multiple methods and metrics under consideration to paint a more holistic picture of the data. At least one method should be a baseline, common approach that may not be optimal for the problem setting, but serves as a helpful comparison.
3.1 Prediction check
Carry out the prediction pipeline, outlined above.
# boxplotsbootstrap_model_errs %>% tidyr::pivot_longer(cols =-c(model, bootstrap_id), names_to ="Metric", values_to ="Value") %>%plotBoxplot(x_str ="Value", y_str ="model", horizontal =FALSE) + ggplot2::facet_wrap(~ Metric) + ggplot2::labs(x ="Method", title ="Validation Prediction Accuracies Over Bootstrapped Training Fits")
3.3 Interpretability
For the models that pass the prediction and stability checks,
Extract the important features in the predictive models that are stable across both data and model perturbations. Determining the importance of a feature can be method dependent.
# TODO: bar plot that works for all modeling backends# plotFeatureImportance(model_results$importance)# vip::vip(model_results$importance,# num_features = 10,# geom = "col") +# simChef::pretty_ggplot_theme()
# TODO: scatter plot that works for all modeling backends# plt <- model_results$importance %>%# tidyr::pivot_wider(names_from = "model", values_from = "Importance") %>%# plotPairs(columns = which(!(colnames(.) %in% "Variable"))) +# ggplot2::theme_bw() # plotly::ggplotly(plt)
3.3.2 With stability
TODO: TT
4 Main Results
Interpret and summarize the prediction and stability results.
Move beyond the global prediction accuracy metrics and dive deeper into individual-level predictions for the validation and/or test set, i.e., provide a more “local” analysis.
Examine any points that had poor predictions.
Examine differences between prediction methods.
TODO: Tiffany - Add examples with interesting observations of prediction accuracy metrics so the user knows what to look for.
